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To Mark losifovich Graev in his 85 birthday 

Let G be a real semisimple Lie group, K its maximal complex subgroup, and Gc 
its complexification. It is known that all the iT-finite matrix elements on G admit 
holomorphic continuation to branching functions on Gc having singularities at the a 
prescribed divisor. We propose a geometric explanation of this phenomenon. 

1 Introduction 

1.1. Casselman theorem. Let G be a real semisimple Lie group, let K be 
the maximal compact subgroup. Let Gc be the complexification of G. 

Let p be an infinite-dimensional irreducible representation of G in a complete 
separable locally convex space W H. Recall that a vector w € W is iC-finite if 
the orbit p{G)v spans a finite dimensional subspace in 

A ii'-finite matrix element is a function on G of the form 

/(ff) = m9)v) 

where d is a ivT- finite vector in W and £ is a finite linear functional, i.e., a 
if-finite element of the dual representation. 

Theorem 1.1 There is an (explicit) complex submanifold A C Gc of codi- 
mension 1 such that each K -finite matrix element of G admits a continuation 
to an analytic multi-valued branching function on Gc \ A. 

Example. Let G = SL(2,M) be the group of real matrices , whose 

d*„„„a„. . 1. Th„ K . S0(2) co„..ts „t „..„cc. f 

\ — sm If cos Lp 



^Supported by the grant FWF, project P19064, Russian Federal Agency for Nuclear Energy, 
Dutch grant NWO. 047.017.015, and grant JSPS-RFBR-07.01.91209 

^the case of unitary representations in Hilbert spaces is sufficiently non-trivial. 

^Let us rephrase the definition. We restrict p to the subgroup K and decompose the 
restriction into a direct sum Vi of finite-dimensional representations of K . Finite sums of 
the form "^^ .^y. Vj are precisely all the A'-finite vectors. 

^ Theorem was obtained in famous preprints of W. Casselman on the Subrepresentation 
Theorem. Unfortunately, these works are unavailable for author; however they are included 
to the paper of W. Casselman and Dr.Milicic [ij. There are (at least) two known proofs; the 
original proof is based on properties of system of partial differential equations for matrix ele- 
ments [l], also the theorem can be reduced to properties of Heckman-Opdam hypergeometric 
functions [2] by a simple trick [3]. 
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where G K; the group Gc is the group of complex 2x2 matrices with de- 
terminant = 1. The submanifold A C SL(2, C) is a union of the foUowing four 
manifolds 

a^O, 5 = 0, c = 0, d^O (1.1) 

Indeed, in this case, there exists a canonical iiT-eigenbasis. All the matrix ele- 
ments in this basis are Gauss hypergeometric functions of the form 

2Fi{a,P;j;9), where 6* = ^ 

be 

where the indices a, f3, 7 depend on parameters of a representation and numbers 
of basis elements (see j6] ) . 

Branching points of 2^1 are = 0, 1, 00. Since ad — be = 1, only 9 = and 
6 = 00 are admissible; this implies (|l.ip . □ 

Thus a representation p of a real semisimple group admits a continuation to 
an analytic matrix- valued function on Gc having singularities at A. This fact 
seems to be strange if we look to explicit constructions of representations. 

Our purpose is to clarify this phenomenon and to find a direct geometric 
construction of the analytic continuation. We achieve this aim for a certain 
special case (namely, for principal maximally degenerate series of SL(n,M), see 
Section 2) and formulate a general conjecture (Section 3). It seems that our 
explanation (a reduction to the 'Thom isotopy Theorem'), see [4], [5]) is trivial. 
However, as far as I know it is not known for experts in the representation 
theory. 

Section 4 is informal and contains a brief exposition of various phenom- 
ena related to analytic continuations of representations. Addendum contains a 
general discussion of holomorphic continuations of representations. 

Acknowledgments. The author is grateful to Alexei Rosly and Victor 
Vasiliev for discussion of this subject. 



2 Isotopy of cycles 

2.1. Principal degenerate series for groups SL(n,M). Let G = SL(n,IR) 
be the group of all real matrices with determinant = 1. The maximal compact 
subgroup K = SO(n) is the group of all real orthogonal matrices. 

Denote by RP"~^ c CP"~^ the real and complex projective spaces; recall 
that the manifold MP""-'^ is orientable iff n is even. 

Denote by dui the SO(n)-invariant Lebesgue measure on RP"~^, let d{uig) 
be its pushforward under the map g, denote by 



J{g,x) := 




the Jacobian of a transformation g at a point x. 
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Fix a E C Define a representation Ta{g) of the group SL(n,R) in the space 
C°°(RP"-i) by the formula 

Ta{g)f{x) = f{xg)J{g,xr 

The representations Tq, are called representations of principal degenerate series. 
If a G 5 + iM., then this representation is unitary in L'^ {WF"~^) . 

2.2. Discriminant submanifold A. Denote by the transpose of a 
matrix g. Denote by A the submanifold in SL(n,C) consisting of matrices g 
such that the equation 

det{gg' - A) = 

has a multiple root. 

We wish to construct a continuation of the function g ^ Ta {g) to a multi- 
valued function on SL(n, C) \ A. 

For simplicity, we assume n is even^ 

2.3. Invariant measure. Denote hy xi : X2 Xn the homogeneous 
coordinates in a projective space. The SO(n)-invariant {n — l)-form on KP""-"^ 
is given by 

duj{x) ~ ^^( — dx\ . . . dxj . . . dxn 

3 3 

This expression can be regarded as a meromorphic (n — l)-form on CP"^^ having 
a pole on the quadric 

Q{x) ■.^Y.^^^Q 

Now we can treat the Jacobian J{g, x) as a meromorphic function on CP"~^ 
having a zero at the quadric Q{x) = and a pole on the shifted quadric Q{gx) = 
0. 

2.4. ivT-finite functions. The following functions span the space of if-finite 
functions on RP"-!; 

Ylx''' 

/(•^) = (^^2)|:fc,/2 ' ^here ^ k, is even 

Evidently, they have singularities at the quadric Q{x) = mentioned above. 

2.5. iiT-finite matrix elements, if-finite matrix elements are given by 
formula 

{/i,/2}= / fi(x)f2{xg)J{g,xrduj{x) (2.1) 

The integrand is a holomorphic form on CP"^^ of the maximal degree ramified 
over quadrics Q{x) — 0, Q{xg) = 0. Denote by il = ii[g] the complement to 

^If n is odd, then we must replace the integrand in 112.11 1 by a form on two sheet covering 
of CP""-*- \ MP""-'-. Also we must replace the cycle RP""-"- by its two-sheet covering. 
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these quadrics. Therefore locally in it the integrand is a closed {n — l)-form. 
Hence we can replace MP"~^ by an arbitrary isotopic cycle C in 11. 

2.6. Reduction to Pham Theorem. Now let g{s) be a path in SL(n, C) 
starting in SL(n,IR). For each s one has a pair Q{x) — 0, Q{x ■ g{s)) = of 
quadrics and the corresponding complement U{g{s)). 

Is it possible to construct an isotopy C{s) of the cycle MP"^^ such that 
C{s) Cil(5(s)) for alls? 

Now recall the following Pham theorem (see F.Pham 0]), V.A.Vasiliev [5]). 

Theorem 2.1 Let Ri{s), Ri{s) be nonsingular complex hypersurfaces in 
CP*^ depending on a parameter. Assume that Rj are transversal (at all points 
for all values of the parameter s ). Then each cycle in the complement to URj (s) 
admits an isotopy according the parameter. 

2.7. Transversality of quadrics. 

Lemma 2.2 Let A, B be non- degenerate symmetric matrices. Assume that all 
the roots of the characteristic equation 

det{A - AB) = 

are pairwise distinct. Then quadrics ^ — and bijXiXj — are 

transversal. 

By the Weierstrass theorem such pair of quadrics can be reduced to 

Y^X.x'^^O, Ea;2 = (2.2) 

where Xj are the roots of the characteristic equation. If they are not transversal 
at a point x, then rank of the matrix 

/ XiXi . . . XnXn\ 
\ Xi ... Xn J 

is 1. Therefore 

{Xi — Xj)xiXj = for all i, j (2-3) 
The system (|2.3p . (12.21) is inconsistent. □ 

2.8. Last step. In our case, the matrices of quadratic forms are gg* and 
1. Therefore, by the virtue of the Pham Theorem a desired isotopy of the cycle 
MP"-i exists. 

3 General case 

By the Subrepresentation Theorem, all the irreducible representations of a 
semisimple group G are subrepresentations of the principal (generally, non- 
unitary) series. Therefore, it suffices to construct analytic continuations for 
representations of the principal series. 
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For definiteness, we discuss the spherical principal series of the group G = 
SL(n,R). 

3.1. Spherical principal series for G = SL(n,M). Denote by F1(M") the 
space of all complete flags of subspaces 

W ; c W^i C • • • C Wn-i C M" 

in M"; here diml/Ffe = k. By Grfe(E") we denote the Grassmannian of all k- 
dimensional subspaces in R". By 7^ we denote the natural projection F1(]R") — > 



By ujk we denote the SO(n)-invariant measure on Grfc(R"). For g £ GL(n, M) 
we denote by Jk{g, V) the Jacobian of the transformation V ^ Vg of Grfc(R"), 

Fix ai, a„_i G C. The representation Ta of the spherical principal 
series of the group SL(n,M) acts in the space C°°(F1(M")) by the formula 

n-l 

Ta{g)f{W) = f{W ■ g) n Jkig^ikmr" 
fc=l 

3.2. Singularities. Consider the symmetric bilinear form in C" given by 

By Lk C Grfe(C") we denote the set of all the fc-dimensional subspaces, where 
the form B is degenerate!!. By C C F1(C") we denote the set of all the flags 
Wi (Z ■ ■ ■ C Wn-1, where Wk € Lk for some k. 

In fact, all the K-tinite functions on F1(R") admit analytic continuations to 
F1(C") \ C {a singularity on £ is a pole or two-sheet branching). 

3.3. A conjecture. 

Conjecture 3.1 Let"f{t) be a path onGL(n, C) avoiding the discriminant sub- 
manifold A, let 7(0) e SL(n,R). Then there is an isotopy G{t) of the cycle 
F1(R") in the space F1(C") avoiding the submanifolds C and C ■ g{s) 

Such isotopy produces an analytic continuation of representations of princi- 
pal series of SL(n, R). 



^Equivalently, we can consider all the (fc — l)-dimensionaI subspaces in CP" ^ tangent to 
the quadric x'^ 
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Addendum. Survey of holomorphic continua- 
tions of representations 

Let G be a connected linear Lie group, i.e., a Lie group admitting an em- 
bedding to a matrix group. Denote by Gc its complexification. Let p be an 
irreducible representation of G (in a Frechet space). We are interested in the 
following problems: 

— Is it possible to extend p holomorphically to Gc? 

— Is it possible to extend p holomorphically to an open domain U C Gc. 
See, also, [TB], Section 1.5. 

A.l. Weyl trick. Let p be a finite dimensional representation of a semisim- 
ple Lie group G. Then p admits the holomorphic continuation to the group Gc- 

A. 2. Why the Weyl trick does not survive for inflnite-dimensional 
unitary representations? Let G be a noncompact Lie group, let p be its 
irreducible faithful unitary representation. Let AT be a noncentral element of the 
Lie algebra g. It is more-or-less obvious that the operator p{X) is unboundecj^. 

Then, for i, s £ R, 

p(exp(t + is)X) = exp(isX) exp(tX) 

propose a collection of arguments that can be combined to a proof in various way. 
First, this is valid for the two-dimensional non-Abelian algebra (and therefore this is valid for 
elements of g that can be included to such a subalgebra). Second, this is valid for nilpotent 
algebras, we apply Kirillov's monomial induction theorem, see [S], then the Lie algebra acts by 
first order differential operators, which are obviously unbounded. Third, for we know that for 
semisimple Lie groups X-spectra are always unbounded. Fourth, elements X with bounded 
p{X) form an ideal. In fifth, we can consider its normalizer 
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Since iX is self-adjoint, then exp{tX) is unitary; on the other hand exp(?sX) 
have to be unbounded for all positive s or for all negative s(and usually it is 
unbounded for all s.. 

However, this argument does not removes completely an idea of holomorphic 
continuation, since it remain two following logical possibilities 

— a holomorphic extension exists in spite of the unboundedness of operators. 

— If a specter of X is contained on a positive half-line, then exp{tX) is 
defined for negative t. We can hope to construct something from elements of 
this kind. 

The second variant is realized for Olshanski semigroups, see below, the first 
variant is general, this follows from the Nelson Theorem. 

A. 3. Nelson's paper. In 1959 E.Nelson [13] proved that each unitary 
irreducible representation p of a real Lie group G has a dense set of analytic 
vectors. This implies that p can be extended analytically to a sufficiently small 
neighborhood of G in Gc- 

However, usually this continuation can be done in a constructive way as it 
is explained below (see also [TB], Section 1.5.). 

A. 4. Induced representations. First, we recall a definition of induced 
representations. 

Consider a Lie group G and its closed connected subgroup H. Let p be a 
representation of if in a finite- dimensional complex space V. 

These data allow to construct canonically a vector bundle {skew product) over 
G/H with a fiber V. Recall a construction (see, for instance, [8]). Consider the 
direct product G x V and the equivalence relation 

{g,v) {gh^^ , p{h)v), where geG,v£V,heH 

Denote by i? = Gxy H the quotient space. The standard map G G/H deter- 
mines a map R G/H (we simply forget v). A fiber can be (noncanonically) 
identified with V. 

Next, the group G acts on G x V by transformations 

(g, v) 1-^ {rg, v), where g£G,v€V,rGG 

This action induces the action oi G on G x^ V, the last action commutes with 
the projection G Xjf V G/H. 

Therefore G acts in the space of sections of the bundle G x h V ~* G / H 
(because graph of a section is a subset in the total space; the group G simply 
move subsets). Induced representation tt — Ind^(p) is the representation of G 
in a space of sections of the bundle G x u V ^ G/H . 

The most important example are principal series, which were partially dis- 
cussed above. 

Our definition is not satisfactory since rather often it is necessary to specify 
the space of sections (for instance, smooth functions, i^-functions, distributions, 
etc.). This discussion is far beyond our purposes, for the moment let us consider 
the space C°°[G/iy;p] of smooth sections. 
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A. 5. Analytic continuation of induced representations. Here we 

discuss some heuristic arguments (see [16j, Section 1.5). Their actual usage 

depends on exphcit situation. 

Denote by H^q the complexification of the group H inside GcH 

The (finite-dimensional) representation p admits a holomorphic extension to 

a representation of the universal covering group of of -f^[C] in the space V. 

For a moment, let us require two assumptionf|j 

— iJ[C] is closed in Gc- 

— p is a linear representation of i?[c] • 

Under these assumptions, the same construction of skew-product produces 
the bundle (Gc) V — > Gc/i?c- Moreover, 

GxhV c (Gc) XHr^V 

Now let us agree with the next assumptiort^. Let the space of holomorphic 
sections of (Gc) V be dense in C°° on G/H . Then we get a holomorphic 
continuation of the induced representation vr to the whole complex group Gc- 
More precisely, we slightly reduce the space of representation, but 'formulae' 
determining a representation are the same. 

This variant is realized for all the nilpotent Lie groups. 

A. 6. Nilpotent Lie groups. 

Example. Let a, 6, c range in K, Consider operators T{a,b,c) in L^(R) 
given by 

T(a,6,c)/(x) = /(x + a)e'''-+^ 

They form a 3-dimensional group, namely the Heisenberg group. Now let a, 
b, c range in C and / ranges in the space Hol{C) of entire functions. Then 
the same formula determines a representation of the complex Heisenberg group 
in Hol{C). After this operation, the space of representation was completely 
changed. However it is easy to invent a dense subspace in L^(IR) consisting of 
holomorphic functions and invariant with respect to all the operators T(a, b, c). 
□ 

Now, let G be a nilpotent Lie group. By the Kirillov Theorem [7], each 
unitary representation of a nilpotent Lie group G is induced from an one- 
dimensional representation of a subgroup H; the manifolds G/H are equivalent 
to standard spaces R™. Therefore, Gc/i?c is the standard complex space C™, 
and we get a representation of Gc in the space of entire functions. 

However, the following R.Goodman-G.L.Litvinov Theorem (R.Goodman [4], 
G.L.Litvinov [11], [12]) is more delicate. 

Theorem. Let p be an irreducible unitary representation of a nilpotent group 
G in a Hilbert space W . There exists a (noncanonical) dense subspace Y with 

*For elements X of the Lie algebra of H we consider the subgroup in Gc spanned by all 
exp((t -I- is)X). 

^The second assumption is very restrictive. It is not hold for the parabolic induction, 
^"if it is not hold, then we go to subsection A. 7, where all the assumptions are omitted. 
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its own Frechet topology and holomorphic representation p of G in the space W 
coinciding with p on G. 

Let us explain how to produce a subspace Y . Let p be a unitary represen- 
tation of a nilpotent group G in a space W . Let / be an entire function on Gc 
(it is specified below). Consider the operator 

p(/)= / f{9)p{9)dg 

JG 

Let r g Gc- We write formally 

P{r)p{f)= I f{g)p{rg)dg= [ f{gr-')p{g) 

JG JG 

Assume that for each r G Gc the function Jrig) ■= f{gf~^) is integrable on G. 
Under this condition we can define operators 

p[r) : |lmage of W 

as just now. 

In fact, we need a subspace Z in L^{G) consisting of entire functions and 
invariant with respect to complex shifts. To be sure that 

Y:==U/ezIm(p(/))cM/ 

is dense, we need in a sequence of positive fj S Z converging to i5-functions; 
then p{fj)v converges to v for all v G W. In what follows we describe a simple 
trick that allows to construct many functions / and a subspace Z . 

First, let G = T„ be the unipotent upper triangular subgroup of order n. 
Let tij, i < j, he the natural coordinates on r„. Write them in the order 

il2, ^23, ^34, • ■ • , t(n-l)n, ^13,^24, • ■ • , t(n-2)n7 *14, ■ • ■ 

and re-denote these coordinates by xi, X2, X3, . . . . In this coordinates, the right 
shift g gr~^ is an affine transformation of the form 

(a;i,X2,X3, . . .) ^ {xi + ai,X2 + 02 + b2iXi,X3 + 03 + b^ixi + b32X2, ■■■) 
Now we can choose a desired function / in the form 

/(x) = exp|— 'y^^PjX^j^' |, where pj > and fci > fc2 > ■ • • > are integers 

For an arbitrary nilpotent G we apply the Ado theorem (in fact, the standard 
proof, see |17p produces a polynomial embedding of G to some r„ with very 
large n). 

A. 7. Local holomorphic continuations. Now \ct G D H be arbitrary. 
The construction of a skew product survives locally. It determines a holo- 
morphic bundle on a (noncanonical) neighborhood U C Gc/Hc oiG/H . Denote 
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by A{U) the space of holomorphic sections of this bundle. Let r G Gc satisfies 
r • G/H C U. The r induces an operator 7r(r) : A{U) — > C°° and we obtain an 
approach to local analytic continuation induced representation. 

A. 8. Local analytic continuations for semisimple groups. For defi- 
niteness, consider G — SL(n, R). Denote by P be the minimal parabolic (i.e., P 
is the group of upper triangular matrices), Then G/ P is the flag space mentioned 
above. Next, Gc — SL(n, C), and Pp] is the group of complex upper-triangular 
matrices of the form 

(bii bi2 --A 
^ ^22 ■ ■ • flj- ^ 0, other Uij are arbitrary. 

Evidently, the group Pc is not simply connected. 

Fix Sj £ C and consider the one-dimensional character 

n 

i=i 

of P. The function Xs is defined on Pc only locally, however this is sufficient 
for the arguments of the previous subsection. 

This kind of arguments can be easily applied to an arbitrary representation 
of a nondegenerate principal series. Keeping in the mind Subquotient Theorem, 
we easily get the following statement. 

Observation 3.1 Let G be a semisimple Lie group. Then there are (non- 
canonical) open sets Ui C U2 CI Gc containing G such that U2 Z) Ui ■ Ui 
and the following property holds. Let p be an irreducible representation of G 
in a Frechet space W. Then there is a (noncanonical) dense subspace Y G W 
(equipped with its own Frechet topology) and an operator-valued holomorphic 
function p : U2 ^ Hom(y, W) .such that p — p on G and 

P{9i)p{92)y = p{9i92)y for gi, g2 e Ui, y £Y 

Certainly, the operators p{g) are unbounded in the topology of llom(VF, W). 

A. 9. Crovirn. D.N.Akhiezer and S.G.Gindikin (see jT]) constructed a cer- 
tain explicit domain A C Gc ('crown') to which all the spherical functions of 
a real semisimple group G can be extended. Also the crown is a domain of 
holomorphy of all irreducible representations of G, see B.Krotz, R.Stanton, [9], 

en]. 

Relation of their constructions with our previous considerations are not com- 
pletely clear. 

A. 10. Olshanski semigroup. In all the previous examples, operators of 
holomorphic continuation arc unbounded in the initial topology. There is an 
important exception. 
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Unitary highest weight representations of a semisimple Lie group G admit 
holomorphic continuations to a certain subsemigroup F C Gc (M.I.Graev [6], 
G.I.Olshanski [12]). Since this situation is well understood, we omit further 
discussion, see also [T5] . 

A.ll. Infinite-dimensional groups. Induction (in different variants) is 
the main tool of construction of representations of Lie groups. 



For infinite-dimensional groups the induction exist but it is a secondary 
tool (however, the algebraic variant of induction is important for infinite dimen- 
sional Lie algebras) . A more effective instrument are symplectic and orthogonal 
spinors. 

Let us realize the standard real orthogonal group 0(2n) as a group of {n + 
n) X (n + n) complex matrices g having the structure 



Actually we do the following. Consider the space R^" equipped with a standard 
basis ei, e„, e„+i,..., e2„. Then we pass to the space C^" and write 
matrices of real orthogonal operators in the basis 

ei -I- icn+i, 62 + ien+2, . . . , e„ + ie2n, ei - ie„+i, 62 - ie„+2, . . . , e„ - ie2„ 

Next, we put n — 00. Denote by 0U(2cxd) the group of all the bounded 
matrices of the same structure satisfying an additional condition: 'J is a Hilbert- 
Schmidt matrix (i.e., the sum ^ IV'fciP of squares of matrix elements is finite). 

By the well-known theorem of F. A. Berezin [2], [3] and D. Shale- W.Stinespring 
PU] . the spinor representation is well-defined on the group OU(2oo). 

Numerous infinite dimensional groups G can be embedded in a natural way 
to 0U(2cx)), after this we can restrict the spinor representation to G. For 
instance, for the loop group C°° {S^ ,S0{2n)] we consider the natural action 
in the space L^(S'^,R^") and define the operator of the complex structure in 
this space via Hilbert transform (see, for instance, |16|). Applying the spinor 
representation, we get the so-called basic representation of the loop group. For 
production of other highest weight representations we apply restrictions and 
tensoring. 

However, the group 0U(2oo) admits a complexification 0GL(2oo, C) con- 



with Hilbert-Schmidt blocks B and C. The spinor representation of OU(2c!o) 
admits a holomorphic continuation to the complex group 0GL(2cx)), see [13], 

^^If a group acts on the space with measure, then it acts in L^. 





that are orthogonal in the following sense 
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[16] , the operators of continuation are unbounded in the initial topology, but are 
bounded on a certain dense Frechet subspace equipped with its own topology. 

This produces highest weight representation of complex loop groups as free 
byproducts (see another approach in R.Goodman, N.Wallach 51). 

More interesting phenomenon arises for the group Diff of diffeomorphisms 
of circle, in this case the analytic continuation exists in spite of nonexistence 
Diffc, see [15]. 
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